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DETAILED ACTION 

1. This action is responsive to communications: RCE amendment filed 12/7/2005. 

2. Claims 1-28 are pending in the case. Claims 1,16, and 22 are independent claims. 

3. The objection to claims 5 and 20 is dropped in response to amendment. 

Claim Rejections - 35 USC §103 

4. The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1 .56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
invention was made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) 
and potential 35 U.S.C. 102(e), (f) or (g) prior art under 35 U.S.C. 103(a). 

5. Claims 1-28 are rejected under 35 U.S.C. 103(a) as being unpatentable over Stubler 
et al. (hereinafter "Stubler"), US 6,804,684 B2 filed 5/7/2001 in view of Toyama, US 
6,816,847 Bl filed 9/23/1999. 
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Regarding independent claim 1, 16, and 22, Stubler teaches actively selecting 
examples of multimedia content to be annotated by a user and accepting input annotations from 
the user for the selected examples in fig. 2, fig. 6-7, col. 3 line 46 - col. 4 line 12, col. 8 lines 18- 
23, and col. 9 line 65 - col. 10 line 18. Stubler teaches propagating the input annotations to other 
instances of multimedia content and storing the input annotations and the propagated annotations 
in fig. 2, col. 3 line 46 - col. 4 line 12, and col. 8 lines 18-55. 

Stubler does not teach actively selecting examples of multimedia content to be annotated 
by a user, wherein the examples of multimedia content are selected based on at least one 
criterion for achieving a maximal disambiguation result. Toyama does teach actively selecting 
examples of multimedia content to be annotated by a user, wherein the examples of multimedia 
content are selected based on at least one criterion for achieving a maximal disambiguation result 
in col. 5 lines 15-46. 

Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards propagating 
annotations from stored images to similar images that have not yet been annotated. Toyama 
teaches in col. 1 line 47 - col. 2 line 14 that it is directed towards propagating annotations from a 
known training set to similar images that do not have any associated annotations. Thus, the field 
of endeavor is very similar for Stubler and Toyama. It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to have combined the teachings of 
Stubler and Toyama to have created the claimed invention. It would have been obvious and 
desirable to have used the active selection techniques disclosed by Toyama in col. 5 lines 15-46 
to have improved the active selection of images to be annotated as taught by Stubler so that a 
maximal disambiguation result could have been achieved as is the reason for a Toyama selection 
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of a wide variety of images in col. 5 lines 30-32 and a wide variety of annotations for the images 
in col. 5 lines 32-46. 

Regarding dependent claims 2 and 17, Stubler teaches wherein the step of actively 
selecting is performed using a selection technique from the group consisting of deterministic and 
probabilistic in col. 4 line 64 - col. 5 line 19. 

Regarding dependent claims 3 and 18, Stubler teaches wherein the step of actively 
selecting which is performed deterministically or probabilistically, is based on explicit models 
and feature proximity/similarity measures, and returns one or more examples of multimedia 
content to be annotated in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 4 and 19, Stubler teaches wherein the step of actively 
selecting, which is performed deterministically or probabilistically, is based on implicit models 
and feature proximity/similarity measures, and returns one or more examples of multimedia 
content to be annotated in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 5 and 20, Stubler teaches wherein an optimization 
criterion for active selection includes one or more criteria selected form the group consisting of: 
information measures and confidence in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 6 and 21, Stubler teaches wherein the multimedia content 
comprises one or more types selected from the group consisting of: images, audio, video, 
graphics, text, multimedia, Web pages, time series data, surveillance data, sensor data, relational 
data, and XML data in col. 3 line 46 - col. 4 line 12. 

Regarding dependent claim 7, Stubler teaches wherein the input annotations are created 
by a user with reference to a vocabulary in col. 3 line 46 - col. 4 line 12 and col. 8 lines 18-23. 
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Regarding dependent claim 8, Stubler teaches wherein the vocabulary contains one or 
more items selected from the group consisting of: terms, concepts, labels, and annotations in col. 
3 line 46 - col. 4 line 12 and col. 8 lines 18-23. 

Regarding dependent claim 9, Stubler teaches wherein the process of creating input 
annotations by the user involves multimodal interaction with the user using graphical, textual, 
and/or speech interface in fig. 2 and col. 3 line 46 - col. 4 line 12. 

Regarding dependent claim 10, Stubler teaches wherein the input annotations are 
created by means of steps selected from the group consisting of: creating new annotations, 
deleting existing annotations, rejects proposed annotations, and modifying annotations in fig. 6-7 
and col. 9 line 65 - col. 10 line 18. 

Regarding dependent claim 11, Stubler teaches wherein the vocabulary is adaptively or 
dynamically organized and/or limited by the system of the user in fig. 6-7 and col. 9 line 65 - 
col. 10 line 18. 

Regarding dependent claim 12, Stubler teaches wherein the multimodal interaction 
involves speech recognition, gaze detection, finger pointing, expression detection, and/or 
effective computing methods for sensing a user's state in fig. 6-7 and col. 9 line 65 - col. 10 line 
18. 

Regarding dependent claim 13, Stubler teaches wherein the determination of the 
propagation of annotations is made deterministically or probabilistically and on the use of 
models for each annotation or for joint annotations in col. 4 line 64 - col. 5 line 19. 
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Regarding dependent claim 14, Stubler teaches wherein the models are created or 
learned automatically or semi-automatically and/or are updated adaptively from interaction with 
the user in fig. 2, col 3 line 46 - col. 4 line 12, and col. 8 lines 18-55. 

Regarding dependent claim 15, Stubler teaches wherein the models are based on 
nearest neighbor voting or variants, parametric or statistical models, expert systems, rule-based 
systems, or hybrid techniques in fig. 2, col. 3 line 46 - col. 4 line 12, and col. 8 lines 18-55. 

Regarding dependent claims 23, 25, and 27, Stubler does not teach wherein that at least 
one criterion includes an ambiguity level of the selected examples. Toyama does teach wherein 
that at least one criterion includes an ambiguity level of the selected examples in col. 5 lines 15 - 
col. 6 line 50. Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards 
propagating annotations from stored images to similar images that have not yet been annotated. 
Toyama teaches in col. 1 line 47 - col. 2 line 14 that it is directed towards propagating 
annotations from a known training set to similar images that have do not have any associated 
annotations. Thus, the field of endeavor is very similar for Stubler and Toyama. It would have 
been obvious to one of ordinary skill in the art at the time the invention was made to have 
combined the teachings of Stubler and Toyama to have created the claimed invention. It would 
have been obvious and desirable to have used the active selection techniques disclosed by 
Toyama in col. 5 lines 15-46 to have improved the active selection of images to be annotated as 
taught by Stubler so that a maximal disambiguation result could have been achieved as is the 
reason for a Toyama selection of a wide variety of images in col. 5 lines 30-32 and a wide 
variety of annotations for the images in col. 5 lines 32-46. 
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Regarding dependent claims 24, 26, and 28, Stubler does not teach wherein the at least 
one criterion includes a confidence level of the selected examples, the confidence level being 
inversely proportional to a distance of a new feature of the selected examples from a separating 
hyperplane in an induced higher dimensional feature space. Toyama discloses the use of support 
vector machines in col. 6 lines 10-50. Toyama discloses wherein the at least one criterion 
includes a confidence level of the selected examples, the confidence level being inversely 
proportional to a distance of a new feature of the selected examples from a separating hyperplane 
in an induced higher dimensional feature space in col 5 lines 15 - col. 6 line 50. 

Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards propagating 
annotations from stored images to similar images that have not yet been annotated. Toyama 
teaches in col. 1 line 47 - col. 2 line 14 that it is directed towards propagating annotations from a 
known training set to similar images that have do not have any associated annotations. Thus, the 
field of endeavor is very similar for Stubler and Toyama. It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to have combined the teachings of 
Stubler and Toyama to have created the claimed invention. It would have been obvious and 
desirable to have used the active selection techniques disclosed by Toyama in col. 5 lines 15-46 
to have improved the active selection of images to be annotated as taught by Stubler so that a 
maximal disambiguation result could have been achieved as is the reason for a Toyama selection 
of a wide variety of images in col. 5 lines 30-32 and a wide variety of annotations for the images 
in col. 5 lines 32-46. 
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Response to Arguments 

6. Applicant's arguments, see page 7, filed 12/7/2005, with respect to the objection of 
claims 5 and 20 for failing to further limit the subject matter of the previous claim have been 
fully considered and are persuasive. The objection of claims 5 and 20 has been withdrawn. 

7. Applicant's arguments filed 12/7/2005 have been fully considered but they are not 
persuasive. Regarding Applicant's arguments in pages 9 and 10 that Stubler and Toyama do not 
teach or suggest all the limitations of independent claim 1, the Examiner respectfully disagrees. 
The Examiner believes Toyama teaches in col. 5 lines 15-46 actively selecting examples of 
multimedia content to be annotated by a user, wherein the examples of multimedia content are 
selected based on at least one criterion for achieving a maximal disambiguation result. Therein 
Toyama shows that selected examples are annotated by a user and wherein the examples provide 
for a achieving maximal disambiguation result. The Examiner also maintains that Toyama does 
teach propagating annotations from a known training set to similar images that do not have 
associated annotations in col. 1 line 47 - col. 2 line 14. The training set is used by the classifier 
and the output of the classifier are annotations for similar images. The Examiner does provide a 
motivation for combining Stubler and Toyama in that maximal disambiguation result could have 
been achieved with Stubler when modified with the teaching of Toyama of selection of a wide 
variety of images in col. 5 lines 30-32 and a wide variety of annotations for the images in col. 5 
lines 32-46. Therefore, the Examiner maintains that Stubler and Toyama teach or suggest all the 
limitations of independent claim 1. The Examiner maintains the rejection of dependent claims 2- 
15 for at least the same reasons regarding independent claim 1. 
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Regarding Applicant's arguments in page 1 1 that Stubler and Toyama do not teach or 
suggest all the limitations of dependent claim 23, the Examiner respectfully disagrees. Within 
the section cited by the Examiner, Toyama teaches wherein the at least one criterion includes an 
ambiguity level of the selected examples in col. 5 lines 42-46 and col 6 lines 19-31. Specifically 
in col. 6 lines 20 and 21, Toyama describes using positive and negative examples with a 
maximum margin, or as claimed an ambiguity level. Therefore, the Examiner maintains that 
Stubler and Toyama in combination teach or suggest all the limitations of dependent claim 23. 

Regarding Applicant's arguments in pages 1 1 and 12 that Stubler and Toyama do not 
teach or suggest all the limitations of dependent claim 24, the Examiner respectfully disagrees. 
Within the section cited by the Examiner, Toyama teaches wherein the at least one criterion 
includes a confidence level being inversely proportional to a distance of a new feature of the 
selected example from a separating hyperplane in an induced higher dimensional feature space in 
col. 5 lines 42-46 and col. 6 lines 19-31. Specifically, Toyama discusses a score, or as claimed a 
confidence, in col. 5 lines 42-46. Toyama also teaches in col. 6 lines 19-31 that the scores are 
inversely proportional distances from a separating hyperplane in an induced higher dimensional 
feature space. Therefore, the Examiner maintains that Stubler and Toyama in combination teach 
or suggest all the limitations of dependent claim 24. 

Regarding Applicant's arguments in pages 12-14 that Stubler and Toyama do not teach or 
suggest all the limitations of independent claim 16, the Examiner respectfully disagrees. The 
Examiner believes Toyama teaches in col. 5 lines 15-46 actively selecting examples of 
multimedia content to be annotated by a user, wherein the examples of multimedia content are 
selected based on at least one criterion for achieving a maximal disambiguation result. Therein 
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Toyama shows that selected examples are annotated by a user and wherein the examples provide 
for a achieving maximal disambiguation result. The Examiner also maintains that Toyama does 
teach propagating annotations from a known training set to similar images that do not have 
associated annotations in col 1 line 47 - col. 2 line 14. The training set is used by the classifier 
and the output of the classifier are annotations for similar images. The Examiner does provide a 
motivation for combining Stubler and Toyama in that maximal disambiguation result could have 
been achieved with Stubler when modified with the teaching of Toyama of selection of a wide 
variety of images in col. 5 lines 30-32 and a wide variety of annotations for the images in col. 5 
lines 32-46. Therefore, the Examiner maintains that Stubler and Toyama teach or suggest all the 
limitations of independent claim 16. The Examiner maintains the rejection of dependent claims 
17-21 for at least the same reasons regarding independent claim 16. 

Regarding Applicant's arguments in page 14 that Stubler and Toyama do not teach or 
suggest all the limitations of dependent claim 25, the Examiner respectfully disagrees. Within 
the section cited by the Examiner, Toyama teaches wherein the at least one criterion includes an 
ambiguity level of the selected examples in col. 5 lines 42-46 and col. 6 lines 19-31. Specifically 
in col. 6 lines 20 and 21, Toyama describes using positive and negative examples with a 
maximum margin, or as claimed an ambiguity level. Therefore, the Examiner maintains that 
Stubler and Toyama in combination teach or suggest all the limitations of dependent claim 25. 

Regarding Applicant's arguments in pages 14 and 15 that Stubler and Toyama do not 
teach or suggest all the limitations of dependent claim 26, the Examiner respectfully disagrees. 
Within the section cited by the Examiner, Toyama teaches wherein the at least one criterion 
includes a confidence level being inversely proportional to a distance of a new feature of the 
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selected example from a separating hyperplane in an induced higher dimensional feature space in 
col 5 lines 42-46 and col. 6 lines 19-31. Specifically, Toyama discusses a score, or as claimed a 
confidence, in col. 5 lines 42-46. Toyama also teaches in col. 6 lines 19-31 that the scores are 
inversely proportional distances from a separating hyperplane in an induced higher dimensional 
feature space. Therefore, the Examiner maintains that Stubler and Toyama in combination teach 
or suggest all the limitations of dependent claim 26. 

Regarding Applicant's arguments in pages 15-17 that Stubler and Toyama do not teach or 
suggest all the limitations of independent claim 22, the Examiner respectfully disagrees. The 
Examiner believes Toyama teaches in col. 5 lines 15-46 actively selecting examples of 
multimedia content to be annotated by a user, wherein the examples of multimedia content are 
selected based on at least one criterion for achieving a maximal disambiguation result. Therein 
Toyama shows that selected examples are annotated by a user and wherein the examples provide 
for a achieving maximal disambiguation result. The Examiner also maintains that Toyama does 
teach propagating annotations from a known training set to similar images that do not have 
associated annotations in col. 1 line 47 - col. 2 line 14. The training set is used by the classifier 
and the output of the classifier are annotations for similar images. The Examiner does provide a 
motivation for combining Stubler and Toyama in that maximal disambiguation result could have 
been achieved with Stubler when modified with the teaching of Toyama of selection of a wide 
variety of images in col. 5 lines 30-32 and a wide variety of annotations for the images in col. 5 
lines 32-46. Therefore, the Examiner maintains that Stubler and Toyama teach or suggest all the 
limitations of independent claim 22. 
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Regarding Applicant's arguments in pages 17 and 18 that Stubler and Toyama do not 
teach or suggest all the limitations of dependent claim 27, the Examiner respectfully disagrees. 
Within the section cited by the Examiner, Toyama teaches wherein the at least one criterion 
includes an ambiguity level of the selected examples in col. 5 lines 42-46 and col. 6 lines 19-31. 
Specifically in col. 6 lines 20 and 21, Toyama describes using positive and negative examples 
with a maximum margin, or as claimed an ambiguity level. Therefore, the Examiner maintains 
that Stubler and Toyama in combination teach or suggest all the limitations of dependent claim 
27. 

Regarding Applicant's arguments in page 18 that Stubler and Toyama do not teach or 
suggest all the limitations of dependent claim 28, the Examiner respectfully disagrees. Within 
the section cited by the Examiner, Toyama teaches wherein the at least one criterion includes a 
confidence level being inversely proportional to a distance of a new feature of the selected 
example from a separating hyperplane in an induced higher dimensional feature space in col. 5 
lines 42-46 and col. 6 lines 19-31. Specifically, Toyama discusses a score, or as claimed a 
confidence, in col. 5 lines 42-46. Toyama also teaches in col. 6 lines 19-31 that the scores are 
inversely proportional distances from a separating hyperplane in an induced higher dimensional 
feature space. Therefore, the Examiner maintains that Stubler and Toyama in combination teach 
or suggest all the limitations of dependent claim 24. 
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Conclusion 



8. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J. Smith whose telephone number is 571-272-4101. The 
examiner can normally be reached on Mondays-Fridays 7:00am-3:30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Heather R. Herndon can be reached on 571-272-4136. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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